<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  
  
  <title>高性能互连网络与拓扑感知算法 | Hexo</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  <meta name="description" content="This page is for Topologies for HPC Networks and Topology-Aware collective communication algorithms.">
<meta property="og:type" content="article">
<meta property="og:title" content="高性能互连网络与拓扑感知算法">
<meta property="og:url" content="https://chudod.gitee.io/myworks/2022/04/25/Topology_Aware_Collectives/index.html">
<meta property="og:site_name" content="Hexo">
<meta property="og:description" content="This page is for Topologies for HPC Networks and Topology-Aware collective communication algorithms.">
<meta property="og:locale" content="en_US">
<meta property="og:image" content="https://note.youdao.com/yws/api/personal/file/WEBc0e3f19d5bb9c51887b22cf2f8c218ba?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
<meta property="og:image" content="https://note.youdao.com/yws/api/personal/file/WEB5d657aab3c672d03c9ff7dc7b7ac1527?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
<meta property="og:image" content="https://note.youdao.com/yws/api/personal/file/WEBedc750c2ece3d90077374a411436e990?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
<meta property="og:image" content="https://note.youdao.com/yws/api/personal/file/WEB7e70f458806199ea4992bf8eb594660b?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
<meta property="og:image" content="https://note.youdao.com/yws/api/personal/file/WEBbaccac337aea7e37f3f4df12205d3db2?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
<meta property="og:image" content="https://note.youdao.com/yws/api/personal/file/WEB4a9b8282c017c3476aa98370093376a7?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
<meta property="article:published_time" content="2022-04-24T16:00:00.000Z">
<meta property="article:modified_time" content="2022-04-28T03:22:49.186Z">
<meta property="article:author" content="John Doe">
<meta property="article:tag" content="并行计算">
<meta property="article:tag" content="集合通信">
<meta property="article:tag" content="高性能计算">
<meta property="article:tag" content="Topology-Aware">
<meta name="twitter:card" content="summary">
<meta name="twitter:image" content="https://note.youdao.com/yws/api/personal/file/WEBc0e3f19d5bb9c51887b22cf2f8c218ba?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe">
  
    <link rel="alternate" href="/myworks/atom.xml" title="Hexo" type="application/atom+xml">
  
  
    <link rel="shortcut icon" href="/myworks/favicon.png">
  
  
    
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/typeface-source-code-pro@0.0.71/index.min.css">

  
  
<link rel="stylesheet" href="/myworks/css/style.css">

  
    
<link rel="stylesheet" href="/myworks/fancybox/jquery.fancybox.min.css">

  
<meta name="generator" content="Hexo 6.2.0"></head>

<body>
  <div id="container">
    <div id="wrap">
      <header id="header">
  <div id="banner"></div>
  <div id="header-outer" class="outer">
    <div id="header-title" class="inner">
      <h1 id="logo-wrap">
        <a href="/myworks/" id="logo">Hexo</a>
      </h1>
      
    </div>
    <div id="header-inner" class="inner">
      <nav id="main-nav">
        <a id="main-nav-toggle" class="nav-icon"></a>
        
          <a class="main-nav-link" href="/myworks/">Home</a>
        
          <a class="main-nav-link" href="/myworks/archives">Archives</a>
        
      </nav>
      <nav id="sub-nav">
        
          <a id="nav-rss-link" class="nav-icon" href="/myworks/atom.xml" title="RSS Feed"></a>
        
        <a id="nav-search-btn" class="nav-icon" title="Search"></a>
      </nav>
      <div id="search-form-wrap">
        <form action="//google.com/search" method="get" accept-charset="UTF-8" class="search-form"><input type="search" name="q" class="search-form-input" placeholder="Search"><button type="submit" class="search-form-submit">&#xF002;</button><input type="hidden" name="sitesearch" value="https://chudod.gitee.io/myworks"></form>
      </div>
    </div>
  </div>
</header>

      <div class="outer">
        <section id="main"><article id="post-Topology_Aware_Collectives" class="h-entry article article-type-post" itemprop="blogPost" itemscope itemtype="https://schema.org/BlogPosting">
  <div class="article-meta">
    <a href="/myworks/2022/04/25/Topology_Aware_Collectives/" class="article-date">
  <time class="dt-published" datetime="2022-04-24T16:00:00.000Z" itemprop="datePublished">2022-04-25</time>
</a>
    
  <div class="article-category">
    <a class="article-category-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/">并行计算</a>►<a class="article-category-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/Topology-Aware-Algorithms/">Topology-Aware Algorithms</a>
  </div>

  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 class="p-name article-title" itemprop="headline name">
      高性能互连网络与拓扑感知算法
    </h1>
  

      </header>
    
    <div class="e-content article-entry" itemprop="articleBody">
      
        <h1 id="Abstract"><a href="#Abstract" class="headerlink" title="Abstract"></a>Abstract</h1><p>拓扑感知(Topology-Aware)的集合通信算法是近些年兴起的一个研究方向，顾名思义，就是将MPI的集合通信算法部署到高性能计算网络上，来充分发挥网络的性能，并且针对不同网络的拓扑结构和协议特点来设计不同的集合通信算法。这一页面列举了最近时髦的一些HPC网络以及对应的拓扑感知算法。在Top 500排行榜中的计算机系统通常采用的网络拓扑结构有Fattree、Torus、Dragonfly，目前，前几名没有用到Dragonfly。</p>
<h1 id="Gragonfly网络"><a href="#Gragonfly网络" class="headerlink" title="Gragonfly网络"></a>Gragonfly网络</h1><h2 id="Introduction"><a href="#Introduction" class="headerlink" title="Introduction"></a>Introduction</h2><p>Gragonfly（蜻蜓）网络是最近非常时髦的一种HPC网络，在2008年提出：<a target="_blank" rel="noopener" href="https://dl-acm-org-s.nudtproxy.yitlink.com/doi/10.1145/1394608.1382129">Technology-Driven, Highly-Scalable Dragonfly Topology</a>。其优点是高度的可伸缩性(high scalability)，低网络直径(low diameter)、成本低(low cost)等等。目前许多HPC系统都使用了Dragonfly网络拓扑。<br>下图是一个9组路由器72个终端节点的Dragonfly网络的例子（图片来源:<a target="_blank" rel="noopener" href="https://ieeexplore-ieee-org-s.nudtproxy.yitlink.com/document/9644896">文献链接</a>）。</p>
<div style="align: center">
<img src="https://note.youdao.com/yws/api/personal/file/WEBc0e3f19d5bb9c51887b22cf2f8c218ba?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe" width="40%" height="40%"/>   

<p>Dragonfly网络分成Group层、System层这两层：</p>
<ul>
<li>Group层：一个Group内一共$a$个交换机（或路由器），$a$个交换机都是全连接的，一个交换机连接着$p$个计算节点，上图中一个路由器连接两个计算节点。上图中的$9$个虚线框代表着$9$个Group层，每个Group层有$4$个全连接的路由器。</li>
<li>System层：一共有$g$个Group，每个Group视为一个虚拟的交换机，Group之间进行全连接，在图中就是$9$个Group进行全连接。组内的每个交换机与其他交换机的连接中，有$a-1$条链路连接到组内的交换机，还有$h&#x3D;\frac{g-1}{a}$条链路是与其他Group之中的交换机相连结。在上图中，每个路由器有$3$条线连接到组内的路由器，还有$2$条连接到了其他Group中的路由器。并且组内的链路(intra-group)的延迟比组间链路(inter-group)的延迟高不少。</li>
</ul>
<h2 id="网络属性与形式化表达"><a href="#网络属性与形式化表达" class="headerlink" title="网络属性与形式化表达"></a>网络属性与形式化表达</h2><p>Gragonfly网络可以用$DF(p,a,h,g)$来形式化表达，例如上图中的例子可以用$DF(p&#x3D;2,a&#x3D;4,h&#x3D;2,g&#x3D;9)$来表达。一个Gragonfly网络最佳的参数配置是：<br>$$a&#x3D;2p&#x3D;2h$$<br>因此可以得到如下的性质：</p>
<ul>
<li>每个交换机端口的数量为$p+h+(a-1)$</li>
<li>每个虚拟交换机（一个$Group$）中的外接端口数量为$ah$，整个网络的全局链路数为$\frac{a\cdot{h}\cdot{g}}{2}$</li>
<li>$Group$的数量为$g&#x3D;ah+1$（由前面的公式$h&#x3D;\frac{g-1}{a}$推导可得）</li>
<li>$DF(p,a,h,g)$网络中一共有$P&#x3D;p\cdot{a}\cdot{g}&#x3D;ap(ah+1)$个节点，因此$DF(n,2n,n,g)$这种最佳配置的网络中的计算节点的数量为$P&#x3D;2p^2(2p^2+1)$</li>
</ul>
<h2 id="Dragonfly网络的拓扑感知集合通信算法"><a href="#Dragonfly网络的拓扑感知集合通信算法" class="headerlink" title="Dragonfly网络的拓扑感知集合通信算法"></a>Dragonfly网络的拓扑感知集合通信算法</h2><p>Dragonfly网络上做集合通信算法的工作不多，论文只有寥寥几篇。最早的论文发表在$\verb+2012 EuroMPI+$上。这几篇论文设计的集合通信算法中，唯独没有$Allreduce$操作，所以我的师兄才抓住了这个点，在Dragonfly网络上设计$\verb+Allreduce+$算法，发了一篇$\verb+ISPA+$的会议论文，在我看来是非常好的想法了。</p>
<h3 id="Collectives-on-Two-Tier-Direct-Networks"><a href="#Collectives-on-Two-Tier-Direct-Networks" class="headerlink" title="Collectives on Two-Tier Direct Networks"></a><a target="_blank" rel="noopener" href="https://link.springer.com/chapter/10.1007/978-3-642-33518-1_12">Collectives on Two-Tier Direct Networks</a></h3><p>该工作在Dragonfly以及IBM PERCS这两个双层的网络上对常规的集合通信操作（包括$\verb+Scatter Gather Allgather Broadcast Reduce-Scatter Reduce+$这6中操作）针对网络拓扑的特点设计了拓扑感知算法。其思想主要是利用了局部链路(Local Link)比全局链路(Global Link)快很多得特点，而且算法设计得比较简单，优化空间很多。</p>
<h1 id="Bcube网络"><a href="#Bcube网络" class="headerlink" title="Bcube网络"></a>Bcube网络</h1><p>Bcube网络拓扑的结构如下所示：</p>
<div style="align: center">
<img src="https://note.youdao.com/yws/api/personal/file/WEB5d657aab3c672d03c9ff7dc7b7ac1527?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe" width="60%" height="60%"/>   

<p>用$Bcube(n,k)$来形式化的表达$Bcube$网络，$n$表示网络中的每个交换机都有$n$个端口，连接到$n$个不同的主机（$n$一般取$2<del>32$），$k$表示每个主机都有$k$个网卡来连接到$k$个不同的交换机（$k$一般取$2</del>4$）。交换机一共有$k$层，所以每个主机的$k$个网卡分别连接到$0, 1, …, k-1$层的交换机。并且主机和主机之间没有直接的链路连接，必须通过交换机连接。下图所表示的就是一个$Bcube(3,2)$的网络（图片来源：<a target="_blank" rel="noopener" href="https://proceedings.neurips.cc/paper/2018/file/f410588e48dc83f2822a880a68f78923-Paper.pdf">BML</a>）。</p>
<div style="align: center">
<img src="https://note.youdao.com/yws/api/personal/file/WEBedc750c2ece3d90077374a411436e990?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe" width="60%" height="60%"/>   

<p>目前，在Bcube上做集合通信算法设计和优化的工作很少，即便有少数几篇相关的，也是在Bcube上做分布式机器学习参数同步算法的研究。由于参数同步操作与$\verb+Allreduce+$操作高度相似，因此在Bcube上设计参数同步算法也就相当于做$\verb+Allreduce+$算法。<br><a target="_blank" rel="noopener" href="https://proceedings.neurips.cc/paper/2018/file/f410588e48dc83f2822a880a68f78923-Paper.pdf">BML:A High-performance, Low-cost Gradient Synchronization Algorithm for DML Training</a>是第一个在Bcube上做分布式机器学习参数同步算法的工作，其实该参数同步算法并不复杂，本质上就是一个$\verb+Reduce+$ + $\verb+Broadcast+$的组合，如下图所示:</p>
<div style="align: center">
<img src="https://note.youdao.com/yws/api/personal/file/WEB7e70f458806199ea4992bf8eb594660b?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe" width="60%" height="60%"/>  

<h1 id="Torus网络"><a href="#Torus网络" class="headerlink" title="Torus网络"></a>Torus网络</h1><p>Torus的英文直译就是环面的意思，在每一维都将该维度坐标相同的计算节点组织成一个绕接的环，图中展示的是一个二维的Torus网络。</p>
<div style="align: center">
<img src="https://note.youdao.com/yws/api/personal/file/WEBbaccac337aea7e37f3f4df12205d3db2?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe" width="30%" height="30%"/> 

<h1 id="Dcell网络"><a href="#Dcell网络" class="headerlink" title="Dcell网络"></a>Dcell网络</h1><p>Dceil的拓扑结构看起来和Dragonfly网络非常相似。</p>
<div style="align: center">
<img src="https://note.youdao.com/yws/api/personal/file/WEB4a9b8282c017c3476aa98370093376a7?method=download&shareKey=4561f9353c0edc6f6b5daa810174cdbe" width="40%" height="40%"/> 
      
    </div>
    <footer class="article-footer">
      <a data-url="https://chudod.gitee.io/myworks/2022/04/25/Topology_Aware_Collectives/" data-id="cl3kznr0s003mj0ukasmpd03f" data-title="高性能互连网络与拓扑感知算法" class="article-share-link">Share</a>
      
      
      
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/myworks/tags/Topology-Aware/" rel="tag">Topology-Aware</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/myworks/tags/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/" rel="tag">并行计算</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/myworks/tags/%E9%9B%86%E5%90%88%E9%80%9A%E4%BF%A1/" rel="tag">集合通信</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/myworks/tags/%E9%AB%98%E6%80%A7%E8%83%BD%E8%AE%A1%E7%AE%97/" rel="tag">高性能计算</a></li></ul>

    </footer>
  </div>
  
    
<nav id="article-nav">
  
    <a href="/myworks/2022/04/28/Triggered_Operation/" id="article-nav-newer" class="article-nav-link-wrap">
      <strong class="article-nav-caption">Newer</strong>
      <div class="article-nav-title">
        
          使用触发操作卸载集合通信
        
      </div>
    </a>
  
  
    <a href="/myworks/2022/04/20/TorstenHeofler/" id="article-nav-older" class="article-nav-link-wrap">
      <strong class="article-nav-caption">Older</strong>
      <div class="article-nav-title">硬核狠人之Torsten Heofler</div>
    </a>
  
</nav>

  
</article>


</section>
        
          <aside id="sidebar">
  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Categories</h3>
    <div class="widget">
      <ul class="category-list"><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/English/">English</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F/">Россия</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%AF%BC%E8%88%AA/">导航</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/">并行计算</a><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/In-Network-Collectives/">In-Network Collectives</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/MPI%E7%A8%8B%E5%BA%8F%E8%AE%BE%E8%AE%A1/">MPI程序设计</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/MPI%E7%A8%8B%E5%BA%8F%E8%AE%BE%E8%AE%A1%E4%B8%8E%E5%B9%B6%E8%A1%8C%E7%AE%97%E6%B3%95/">MPI程序设计与并行算法</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/MPI%E8%BD%AF%E4%BB%B6%E6%A0%88%E8%AE%BE%E8%AE%A1/">MPI软件栈设计</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/MPI%E9%9B%86%E5%90%88%E9%80%9A%E4%BF%A1%E7%AE%97%E6%B3%95/">MPI集合通信算法</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/Topology-Aware-Algorithms/">Topology-Aware Algorithms</a></li></ul></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E6%95%B4%E6%B4%BB%E7%B3%BB%E5%88%97/">整活系列</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E7%A1%AC%E6%A0%B8%E7%8B%A0%E4%BA%BA/">硬核狠人</a></li><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E8%AE%A1%E7%AE%97%E6%9C%BA%E5%9F%BA%E7%A1%80%E7%9F%A5%E8%AF%86/">计算机基础知识</a><ul class="category-list-child"><li class="category-list-item"><a class="category-list-link" href="/myworks/categories/%E8%AE%A1%E7%AE%97%E6%9C%BA%E5%9F%BA%E7%A1%80%E7%9F%A5%E8%AF%86/%E6%93%8D%E4%BD%9C%E7%B3%BB%E7%BB%9F/">操作系统</a></li></ul></li></ul>
    </div>
  </div>


  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Tags</h3>
    <div class="widget">
      <ul class="tag-list" itemprop="keywords"><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/English-Vocabulary/" rel="tag">English Vocabulary</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/English-Writing/" rel="tag">English Writing</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/MPI/" rel="tag">MPI</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/Topology-Aware/" rel="tag">Topology-Aware</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F/" rel="tag">Россия</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E5%9C%A8%E7%BD%91%E8%AE%A1%E7%AE%97/" rel="tag">在网计算</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E5%B9%B6%E8%A1%8C%E7%AE%97%E6%B3%95/" rel="tag">并行算法</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/" rel="tag">并行计算</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E6%93%8D%E4%BD%9C%E7%B3%BB%E7%BB%9F/" rel="tag">操作系统</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E6%95%B4%E6%B4%BB/" rel="tag">整活</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E7%94%9F%E6%B4%BB/" rel="tag">生活</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E7%A1%AC%E6%A0%B8%E7%8B%A0%E4%BA%BA/" rel="tag">硬核狠人</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E9%9B%86%E5%90%88%E9%80%9A%E4%BF%A1/" rel="tag">集合通信</a></li><li class="tag-list-item"><a class="tag-list-link" href="/myworks/tags/%E9%AB%98%E6%80%A7%E8%83%BD%E8%AE%A1%E7%AE%97/" rel="tag">高性能计算</a></li></ul>
    </div>
  </div>


  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Tag Cloud</h3>
    <div class="widget tagcloud">
      <a href="/myworks/tags/English-Vocabulary/" style="font-size: 10px;">English Vocabulary</a> <a href="/myworks/tags/English-Writing/" style="font-size: 10px;">English Writing</a> <a href="/myworks/tags/MPI/" style="font-size: 17.5px;">MPI</a> <a href="/myworks/tags/Topology-Aware/" style="font-size: 10px;">Topology-Aware</a> <a href="/myworks/tags/%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F/" style="font-size: 12.5px;">Россия</a> <a href="/myworks/tags/%E5%9C%A8%E7%BD%91%E8%AE%A1%E7%AE%97/" style="font-size: 12.5px;">在网计算</a> <a href="/myworks/tags/%E5%B9%B6%E8%A1%8C%E7%AE%97%E6%B3%95/" style="font-size: 10px;">并行算法</a> <a href="/myworks/tags/%E5%B9%B6%E8%A1%8C%E8%AE%A1%E7%AE%97/" style="font-size: 20px;">并行计算</a> <a href="/myworks/tags/%E6%93%8D%E4%BD%9C%E7%B3%BB%E7%BB%9F/" style="font-size: 15px;">操作系统</a> <a href="/myworks/tags/%E6%95%B4%E6%B4%BB/" style="font-size: 10px;">整活</a> <a href="/myworks/tags/%E7%94%9F%E6%B4%BB/" style="font-size: 10px;">生活</a> <a href="/myworks/tags/%E7%A1%AC%E6%A0%B8%E7%8B%A0%E4%BA%BA/" style="font-size: 10px;">硬核狠人</a> <a href="/myworks/tags/%E9%9B%86%E5%90%88%E9%80%9A%E4%BF%A1/" style="font-size: 15px;">集合通信</a> <a href="/myworks/tags/%E9%AB%98%E6%80%A7%E8%83%BD%E8%AE%A1%E7%AE%97/" style="font-size: 12.5px;">高性能计算</a>
    </div>
  </div>

  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Archives</h3>
    <div class="widget">
      <ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="/myworks/archives/2022/05/">May 2022</a></li><li class="archive-list-item"><a class="archive-list-link" href="/myworks/archives/2022/04/">April 2022</a></li><li class="archive-list-item"><a class="archive-list-link" href="/myworks/archives/2022/03/">March 2022</a></li></ul>
    </div>
  </div>


  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Recent Posts</h3>
    <div class="widget">
      <ul>
        
          <li>
            <a href="/myworks/2022/05/24/Russian_Grammar/">Русская Граммаатика(俄语语法)</a>
          </li>
        
          <li>
            <a href="/myworks/2022/05/18/OS_FileSystem/">文件系统概念和Ext2文件系统</a>
          </li>
        
          <li>
            <a href="/myworks/2022/05/14/English_Sentences/">English Sentences</a>
          </li>
        
          <li>
            <a href="/myworks/2022/05/11/English_Vocabulary/">English Vocabulary</a>
          </li>
        
          <li>
            <a href="/myworks/2022/05/11/Russian_Vocabulary/">Русские Слова(俄语单词)</a>
          </li>
        
      </ul>
    </div>
  </div>

  
</aside>
        
      </div>
      <footer id="footer">
  
  <div class="outer">
    <div id="footer-info" class="inner">
      
      &copy; 2022 John Doe<br>
      Powered by <a href="https://hexo.io/" target="_blank">Hexo</a>
    </div>
  </div>
</footer>

    </div>
    <nav id="mobile-nav">
  
    <a href="/myworks/" class="mobile-nav-link">Home</a>
  
    <a href="/myworks/archives" class="mobile-nav-link">Archives</a>
  
</nav>
    


<script src="/myworks/js/jquery-3.4.1.min.js"></script>



  
<script src="/myworks/fancybox/jquery.fancybox.min.js"></script>




<script src="/myworks/js/script.js"></script>





  </div>
</body>
</html>